Image processing device and image processing program for determining similarity factors of pixels

ABSTRACT

An image processing device according to the present invention includes a similarity factor calculating portion locally comparing pixels that compose an image so as to calculate a similarity factor of the pixels in a predetermined direction, and includes an analyzing portion analyzing the structure of the image in accordance with the similarity factor calculated by the similarity factor calculating portion. In particular, the similarity factor calculating portion selects and changes pixels to be compared in accordance with color shift of the image to calculate the similarity factor. As a result, a similarity factor can be calculated with suppressing influence of color shift.

This Application is a 371 of PCT/JP02/13801 filed Dec. 27, 2002.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing device forperforming an image process and to an image processing program forcausing a computer to perform an image process.

2. Description of the Related Art

Nowadays, an image processing device that performs the following imageprocesses for image data photographed by an image sensor, a filmscanner, or the like has been disclosed.

-   (1) a process for transforming input image data into new image data    (for example, an interpolating process, a color interpolating    process, a color coordinate transforming process, an edge    emphasizing process, a resolution transforming process, a spatial    frequency filtering process, and so forth).-   (2) an image analyzing process for analyzing an image structure of    image data (for example, a process for determining local similarity    in image data and analyzing local image structure in accordance with    the determined result).

In particular, the foregoing image analyzing process is an importanttechnology for performing an advanced interpolating process.

On the other hand, in an optical system of an electronic camera, aphenomenon is known that slight color shift takes place in image datadue to chromatic aberration of magnification. In recent years, sincephotographic resolutions of electronic cameras have become higher, evenslight color shift tends to be unignorable.

It is known that when a film that has chromatic aberration ofmagnification is read by a film scanner, there occurs a phenomenon thatslight color shift takes place in image data.

As countermeasures against such color shift due to chromatic aberrationof magnification, a correction technology for enlarging or reducing animage plane of each color component has been disclosed (hereinafterreferred to as “correction of color shifted image”).

SUMMARY OF THE INVENTION

The foregoing image analyzing process has high possibility that colorinformation of desired positions cannot be compared due to influence ofcolor shift. Thus, there occurs a risk that the determination ofsimilarity has an error.

Such a determination error of similarity might disturb an advancedinterpolating process. Particularly, in a color interpolating process ofa 1-CCD type electronic camera, a determination error of similaritywould result in false colors.

To solve such a problem, after the foregoing color shifted image iscorrected, an advanced color interpolating process may be performed inconsideration of similarity. However, since such a method requires twointerpolating processes, image quality will be largely deteriorated. Inaddition, the two interpolating processes require a large buffer memoryspace for the image, and the calculating process will take a long time.

To solve the foregoing problems, an object of the present invention isto properly determine similarity factors of color shifted image data.

Moreover, another object of the present invention is to properly performan image processing on color shifted image data.

Next, the present invention will be described.

An image processing device according to the present invention includes asimilarity factor calculating portion locally comparing pixels thatcompose an image so as to calculate a similarity factor of the pixels ina predetermined direction, and an analyzing portion analyzing structureof the image in accordance with the similarity factor calculated by thesimilarity factor calculating portion. In particular, the similarityfactor calculating portion calculates the similarity factor of thepixels in accordance with positions of the pixels in the image.

Preferably, in the similarity factor calculating portion, the similarityfactor includes a value determined by performing a weighted addition of“a similarity factor element in same colors obtained by comparing pixelsof same colors” and “a similarity factor element in different colorsobtained by comparing pixels of different colors”. In this case, thesimilarity factor calculating portion preferably changes a weightingcoefficient for the weighted addition in accordance with the positionsof the pixels whose similarity factor is to be calculated.

Still preferably, the similarity factor calculating portion changes theweighting coefficient in accordance with distance from “the center ofthe image” to “the pixels whose similarity factor is to be calculated”.

In addition, the similarity factor calculating portion preferablychanges the weighting coefficient so that weight of the similarityfactor element in different colors is smaller at a peripheral part ofthe image than at a center part of the image.

In addition, the similarity factor calculating portion preferablydivides the image into a plurality of regions and changes the weightingcoefficient for each of the divided regions.

In addition, the similarity factor calculating portion preferablychanges the weighting coefficient in accordance with a characteristic ofcolor shift of an optical system for generating the image.

In addition, the similarity factor calculating portion preferablycalculates a similarity factor including a weighted addition value of “asimilarity factor element in same colors obtained by comparing pixels ofsame colors” and “a similarity factor element in different colorsobtained by comparing pixels of different colors” in a region containingthe center of the image (referred to as center region).

In addition, the similarity factor calculating portion preferablycalculates a similarity factor that does not contain the similarityfactor element in different colors, but contains the similarity factorelement in same colors in a region other than the center region.

In addition, the similarity factor calculating portion preferablychanges relative location of “a considered pixel whose similarity factoris to be calculated” and “a reference pixel that is referenced whencalculating the similarity factor” in accordance with position of theconsidered pixel.

In addition, the similarity factor calculating portion preferablychanges the relative location in accordance with distance from thecenter of the image to the considered pixel.

In addition, the similarity factor calculating portion preferablydivides the image into a plurality of regions and changes the relativelocation for each of the regions.

In addition, the similarity factor calculating portion preferablychanges the relative location in accordance with a characteristic ofcolor shift of an optical system for generating the image.

Another image processing device according to the present inventionincludes a similarity factor calculating portion locally comparingpixels that compose an image so as to calculate a similarity factor ofthe pixels in a predetermined direction, and includes a pixel valuecalculating portion referencing the pixels that compose the image so asto obtain a value of a new pixel. In this case, the similarity factorcalculating portion calculates the similarity factor of the pixels inaccordance with positions of the pixels in the image. The pixel valuecalculating portion changes relative location of “a considered pixelfrom which the new value is to be calculated” and “a reference pixelthat is referenced when calculating the new value” in accordance withposition of the considered pixel and the similarity factor, andcalculates the new value of the considered pixel.

In addition, the pixel value calculating portion preferably changes therelative location predetermined in accordance with “distance from thecenter of the image to the considered pixel”, according to thesimilarity factor.

In addition, the pixel value calculating portion preferably divides theimage into a plurality of regions and changes the relative locationpredetermined for each of the regions in accordance with the similarityfactor.

In addition, the pixel value calculating portion preferably changes therelative location predetermined in accordance with “a characteristic ofcolor shift of an optical system for generating the image”, according tothe similarity factor.

In addition, when the new value contains an element composed of a valueof a color that has color shift and the element is calculated, the pixelvalue calculating portion preferably selects, as the reference pixel, apixel in a local region selected in accordance with a direction of thecolor shift and an amount of the color shift. In this case, the pixelvalue calculating portion preferably adds with weight the selectedreference pixel in accordance with the similarity factor so as tocalculate the new value.

An image processing program according to the present invention causes acomputer to perform functions of a similarity factor calculating portionlocally comparing pixels that compose an image so as to calculate asimilarity factor of the pixels in a predetermined direction, and of ananalyzing portion analyzing structure of the image in accordance withthe similarity factor calculated by the similarity factor calculatingportion. In this case, the similarity factor calculating portionrealized on the computer calculates the similarity factor of the pixelsin accordance with positions of the pixels in the image.

Another image processing program according to the present inventioncauses a computer to perform functions of a similarity factorcalculating portion locally comparing pixels that compose an image so asto calculate a similarity factor of the pixels in a predetermineddirection, and of a pixel value calculating portion referencing thepixels that compose the image so as to obtain a value of a new pixel. Inthis case, the similarity factor calculating portion realized on thecomputer calculates the similarity factor of the pixels in accordancewith positions of the pixels in the image. The pixel value calculatingportion changes relative location of “a considered pixel from which thenew value is to be calculated” and “a reference pixel that is referencedwhen calculating the new value” in accordance with position of theconsidered pixel and the similarity factor, and calculates the new valueof the considered pixel.

BRIEF DESCRIPTION OF DRAWINGS

The features and advantages of the present invention will becomeapparent from the following description:

FIG. 1 is a block diagram showing an electronic camera;

FIG. 2 is a schematic diagram showing an example of an arrangement ofcolor components of image data;

FIG. 3 is a schematic diagram showing the relation of a distance Rt andthreshold values r0 and r1;

FIG. 4 is a schematic diagram showing a characteristic of a color shiftamount;

FIG. 5 is a schematic diagram showing an example of divided blocks;

FIG. 6 is a schematic diagram showing an example of color shift;

FIG. 7 is a schematic diagram showing examples of coefficient filters;

FIG. 8 is a schematic diagram showing examples of coefficient filters;

FIG. 9 is a schematic diagram showing examples of coefficient filters;and

FIG. 10 is a schematic diagram showing examples of coefficient filters.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

First of all, the structure of an electronic camera that is in commonwith each of embodiments of the present invention will be described. Animage processing device according to the present invention correspondsto an image processing portion that is built in the electronic camera.

FIG. 1 is a block diagram showing an electronic camera.

In FIG. 1, the electronic camera 1 includes an A/D converting portion10, an image processing portion 11, a controlling portion 12, a memory13, a compressing/decompressing portion 14, and a display imagegenerating portion 15. In addition, the electronic camera 1 includes amemory card interface portion 17 and an external interface portion 18.The memory card interface portion 17 realizes an interface with a memorycard (card shaped removable memory) 16. The external interface portion18 realizes an interface with an external device (for example, apersonal computer and so forth) through a predetermined cable or awireless transmission path. These structural elements are mutuallyconnected through a bus.

Furthermore, the electronic camera 1 includes a photographic opticalsystem 20, an image sensor 21, an analog signal processing portion 22,and a timing controlling portion 23.

An optical image is focused on the image sensor 21 through thephotographic optical system 20. An output of the image sensor 21 isconnected to the analog signal processing portion 22. An output of theanalog signal processing portion 22 is connected to the A/D convertingportion 10. An output of the controlling portion 12 is connected to thetiming controlling portion 23. An output of the timing controllingportion 23 is connected to the image sensor 21, the analog signalprocessing portion 22, the A/D converting portion 10, and the imageprocessing portion 11.

In addition, the electronic camera 1 includes an operating portion 24and a monitor 25. The operating portion 24 corresponds to a releasebutton, a mode selection button, and so forth.

An output of the operating portion 24 is connected to the controllingportion 12. An output of the display image generating portion 15 isconnected to the monitor 25.

When the operator selects a photographing mode and presses the releasebutton of the operating portion 24 of the electronic camera 1 structuredas shown in FIG. 1, the controlling portion 12 controls the image sensor21, the analog signal processing portion 22, and the A/D convertingportion 10 through the timing controlling portion 23. As a result, theimage sensor 21 generates an image signal in accordance with an opticalimage. The analog signal processing portion 22 performs a signal processsuch as a black level compensation for the image signal. The A/Dconverting portion 10 digitizes the image signal and supplies theresultant image data to the image processing portion 11.

A color filter array (not shown) in which three color filters of R, G,and B (red, green, and blue) are arranged in Bayer-pattern is disposedon the image sensor 21. Thus, image data supplied to the imageprocessing portion 11 is data having one of color components of R, G,and B in units of one pixel.

FIG. 2 is a schematic diagram showing an example of an arrangement ofcolor components of such image data.

FIG. 2 shows types of color components with R, G, and B, and positionsof pixels corresponding to respective color components with coordinates[X, Y]. In calculation formulas that follow, when green components ofcolor information in individual pixels are distinguished from othercolor components, R and B shown in FIG. 2 are substituted with Z. Colorinformation of pixels corresponding to red or blue components isrepresented by Z[i, j]. Color information of pixels corresponding togreen components is represented by G[i, j].

The image processing portion 11 is realized by hardware such as an ASIC(Application Specific IC) or a pre-recorded image processing program(that corresponds to an image processing program described in Claims).The image processing portion 11 performs various image processes forimage data arranged as shown in FIG. 2.

However, color shift has been taken place in image data supplied to theimage processing portion 11 due to chromatic aberration of magnificationof the photographic optical system 20. The image processing portion 11performs an image process in consideration of a characteristic of thecolor shift.

When necessary, image data that has been subjected to the image processby the image processing portion 11 is compressed in a predeterminedmanner by the compressing/decompressing portion 14. The compressed imagedata is recorded in the memory card 16 through the memory card interfaceportion 17.

Alternatively, the image data having been subjected to the image processby the image processing portion 11 may be directly recorded in thememory card 16 or supplied to an external device through the externalinterface portion 18 without being compressed.

When the operator selects a displaying mode with the operating portion24, image data recorded in the memory card 16 is read therefrom throughthe memory card interface portion 17 and decompressed by thecompressing/decompressing portion 14. The decompressed image data isdisplayed on the monitor 25 through the display image generating portion15. Alternatively, the decompressed image data may not be displayed onthe monitor 25, but supplied to an external device through the externalinterface portion 18.

Next, image processes of the image processing portion 11 that is one ofthe features of the present invention will be described according toeach embodiment thereof. In first and second embodiments, a process forcalculating a similarity factor of each pixel in a predetermineddirection (hereinafter referred to as “similarity factor calculatingprocess”) will be described among the image analyzing processesperformed by the image processing portion 11. In a third embodiment, animage process for correlating a Y component to all the pixels(hereinafter referred to as “Y component generating process”) will bedescribed among color coordinates transformation process for generatingan image data of YCbCr color coordinates from an image data arranged asshown in FIG. 2.

Description of First Embodiment

Next, the similarity factor calculating process according to the firstembodiment will be described.

First of all, the image processing portion 11 calculates similarityfactor components of plural types in a plurality of predetermineddirections, for each of considered pixels to be calculated. The pluraltypes of similarity factor components contain similarity factorcomponents that compose a similarity factor element in same colors (thatare absolute values of difference between color information of samecolors and that correspond to a similarity factor component of RR, asimilarity factor component of GG, and a similarity factor component ofBB that will be described later), similarity factor components thatcompose a similarity factor element in different colors (that areabsolute values of difference between color information of differentcolors and that correspond to a similarity factor component of RG and asimilarity factor component of GB that will be described later), and soforth.

When a pixel that is placed at coordinates [i, j] and that correspondsto a red component is a considered pixel, a plurality of similarityfactor components in the vertical direction can be calculated withreference to color information and luminance (values calculated by laterdescribed Formula 7) of pixels arranged in the vertical direction, byFormulas 1 to 6 that follow. Each similarity factor component calculatedhere represents higher similarity, as the value gets smaller.

Similarity factor component of RR in vertical direction:Cvr[i, j]=(|Z[i, j−2]−Z[i, j]|+|Z[i, j+2]−Z[i, j]|)/2  Formula 1

Similarity factor component of GG in vertical direction:Cvg[i, j]=|G[i, j−1]−G[i, j+1]|  Formula 2

Similarity factor component of BB in vertical direction:Cvb[i, j]=(|Z[i−1, j−1]−Z[i−1, j+1]|+|Z[i+1, j−1]−Z[i+1,j+1]|)/2  Formula 3

Similarity factor components of RG in vertical direction:Cvrg[i, j]=(|G[i, j−1]−Z[i, j]|+|G[i, j+1]−Z[i, j]|)/2  Formula 4

Similarity factor component of GB in vertical direction:Cvgb[i, j]=(|Z[i−1, j−1]−G[i−1, j]|+|Z[i−1, j+1]−G[i−1, j]|+|Z[i+1,j−1]−G[i+1, j]|+|Z[i+1, j+1]−G[i+1,j]|)/4  Formula 5

Similarity factor component in luminance in vertical direction:Cvy[i, j]=(|Y|[i, j−1]−Y[i, j]|+|Y[i, j+1]−Y[i, j]|)/2  Formula 6

In Formula 6, Y[i, j] is a value calculated by Formula 7. Y[i, j]corresponds to luminance that is generated by a filtering process foraveraging color information of pixels arranged around a target pixelfrom which a similarity factor component is calculated in the ratio ofR:G:B=1:2:1. Note that in Formula 7, A represents any color informationthat is the value of G or Z depending on the position.Y[i, j]=(4·A[i, j]+2·(A[i, j−1]+A[i, j+1]+A[i−1, j]+A[i+1, j])+A[i−1,j−1]+A[i−1, j+1]+A[i+1, j−1]+A[i+1, j+1])/16  Formula 7

In the forgoing example, a plurality of similarity factor components inthe vertical direction of a pixel corresponding to a red component wascalculated. A plural types of similarity factor components in anotherdirection and similarity factor components of pixels corresponding toother color components can also be calculated in the same manner.

In the forgoing example, similarity factors in the vertical directionwere calculated. Likewise, similarity factors in the horizontaldirection can be calculated.

Next, the image processing portion 11 adds with weight a plurality oftypes of similarity factor components in each direction for eachconsidered pixel with weighting coefficients that allow the ratio of thesimilarity factor elements in same colors to the similarity factorelements in different colors to be varied in accordance with thedistance from the center of the image to a considered pixel so as toobtain the similarity factor of the considered pixel.

For example, the image processing portion 11 adds with weight aplurality of similarity factor components in the vertical directioncalculated in the above manner in accordance with the following Formula8, and calculates similarity factors in the vertical direction.

$\quad\begin{matrix}\begin{matrix}{{{Cv}\left\lbrack {i,j} \right\rbrack} = {k\; 1\left( {{{a1} \cdot {{Cvr}\left\lbrack {i,j} \right\rbrack}} + {a\;{2 \cdot {{Cvg}\left\lbrack {i,j} \right\rbrack}}} + {a\;{3 \cdot}}} \right.}} \\{\left. {{Cvb}\left\lbrack {i,j} \right\rbrack} \right) + {k\; 2\left( {{{b1} \cdot {{Cvrg}\left\lbrack {i,j} \right\rbrack}} + {b\;{2 \cdot}}} \right.}} \\{\left. {{Cvgb}\left\lbrack {i,j} \right\rbrack} \right) + {c \cdot {{Cvy}\left\lbrack {i,j} \right\rbrack}}}\end{matrix} & {{Formula}\mspace{20mu} 8}\end{matrix}$

where k1, a1, a2, a3, k2, b1, b2, and c in Formula 8 are predeterminedvalues that satisfy the following Condition 1.k1(a1+a2+a3)+k2(b1+b2)+c=1  Condition 1

where the ratio of k1 to K2 is predetermined so that the value of k2 isreduced as the distance from the center of the image to the consideredpixel becomes longer.

Threshold values r0 and r1 are predetermined in accordance with therelation of a distance Rt from the center of the image to a consideredpixel, color shift due to information that represents a characteristicof chromatic aberration of magnification of the photographic opticalsystem 20 (for example, a lens type, a focal distance, an aperturevalue, and so forth), and a pixel pitch. With the threshold values r0and r1, the ratio of k1 to k2 can be obtained as follows.

when Rt≦r0, k1:k2=2:2,

when r0<Rt≦r1, k1:k2=2:1, and

when r1<Rt, k1:k2=2:0.

When the color shift amount due to the chromatic aberration ofmagnification of the photographic optical system 20 represents thecharacteristic as shown in FIG. 4, the relations of the threshold valuesmay be r0=L0 and r1=L1.

As described above, according to Formula 8, a similarity factor in thevertical direction is calculated by adding with weight a similarityfactor element in same colors, a similarity factor element in differentcolors, and a similarity factor component in luminance in the verticaldirection. The similarity factor element in same colors is calculated byadding with weight a similarity factor component of RR, a similarityfactor component of GG, and a similarity factor component of BB in thevertical direction. The similarity factor element in different colors iscalculated by adding with weight a similarity factor component of RG anda similarity factor component of GB in the vertical direction.

In addition, according to Formula 8, the ratio of the similarity factorelement in same colors to the similarity factor element in differentcolors is determined in accordance with the ratio of k1 to k2. Thus, thesimilarity factor in the vertical direction is calculated so that theweight of the similarity factor element in different colors is smallerat the peripheral part of the image than at the center part of theimage. In particular, when a pixel whose similarity factor is to becalculated is placed at a position shown in FIG. 3 (at coordinates [i,j]) (r1<Rt), the similarity factor in the vertical direction of thepixel does not contain a similarity factor element in different colors,but contains a similarity factor element in same colors.

In other words, according to the first embodiment, when the color shiftamount due to the chromatic aberration of magnification of thephotographic optical system 20 increases as the distance from the centerof the image gets longer as shown in FIG. 4, the weight to thesimilarity factor element in different colors that is affected by thecolor shift can be decreased as the color shift amount is increased.

Thus, according to the first embodiment, the similarity factor can beaccurately calculated with suppressing the influence of the color shiftthat varies depending on the position of a considered pixel.

Description of Second Embodiment

Next, a similarity factor calculating process according to the secondembodiment will be described.

Like the first embodiment, first of all, the image processing portion 11calculates plural types of similarity factor components for eachconsidered pixel in a plurality of predetermined directions.

Thereafter, the image processing portion 11 adds with weight pluraltypes of similarity factor components for each considered pixel in eachof the directions, and calculates the similarity factor. However, theimage processing portion 11 divides image data supplied through the A/Dconverting portion 10 and so forth into a predetermined number ofblocks. When the image processing portion 11 calculates similarityfactors of pixels in the same block, it uses a weighting coefficientthat causes the ratio of a similarity factor element in same colors to asimilarity factor element in different colors to be the same. The imageprocessing portion 11 changes the weighting coefficient in accordancewith the distance from the center of the image to each block.

For example, when image data is divided into 5×7 blocks as shown in FIG.5, and a plurality of similarity factor components in the verticaldirection calculated in accordance with Formula 1 to Formula 6 accordingto the first embodiment are added with weight in accordance with Formula8, the ratio of K1 to K2 that represents the ratio of a similarityfactor element in same colors to a similarity factor element indifferent colors may have the following values.

In block 1: “k1:k2=2:2”.

In block 2 to block 9: “k1:k2=2:1”.

In block 10 to block 35: “k1:k2=2:0”.

In other words, the ratio of k1 to k2 is determined so that the value ofk2 of a block is smaller at the peripheral part of the image than at thecenter part of the image.

Thus, a similarity factor in the vertical direction is calculated sothat the weight of a similarity factor element in different colors issmaller at the peripheral part of the image than at the center part ofthe image. In particular, a similarity factor of pixels in the verticaldirection of the block 10 to block 35 does not contain a similarityfactor element in different colors, but contain a similarity factorelement in same colors.

As described above, according to the second embodiment, the weight of asimilarity factor element in different colors that is affected by colorshift can be decreased as the color shift is increased, likewise in thefirst embodiment. As a result, a similarity factor can be accuratelycalculated with suppressing influence of color shift that variesdepending on the position of a considered pixel.

In addition, according to the second embodiment, it is not necessary tocalculate the distance from the center of the image to a consideredpixel, unlike the first embodiment. Thus, a similarity factor can beaccurately and quickly calculated.

Description of Third Embodiment

Next, a Y component generating process according to the third embodimentwill be described.

First of all, the image processing portion 11 calculates a color shiftamount (a color shift amount of a red component to a green component anda color shift amount of a blue component to a green component) for eachconsidered pixel from which a Y component is to be generated, based oninformation (for example, a lens type, a focal distance, an aperturevalue, and so forth) that represents a characteristic of chromaticaberration of magnification of the photographic optical system 20, andbased on the distance from the center of the image to the consideredpixel and the direction. However, it is assumed that the color shiftamount here is calculated in consideration of the direction of the colorshift.

The image processing portion 11 may pre-record a look up table (LUT)that correlates the distance from the center of an image to a consideredpixel, direction, and color shift amount in accordance with informationrepresenting a characteristic of chromatic aberration of magnificationof the photographic optical system 20, to determine the color shiftamount with reference to the LUT.

Thereafter, the image processing portion 11 calculates color informationof a green component of a considered pixel with reference to colorinformation of green components in a local region (for example, 4×4pixels) containing the considered pixel.

For example, when a pixel corresponding to a red component (or a bluecomponent) is a considered pixel, color information of a green componentof the considered pixel can be calculated by averaging color informationof green components placed at the immediately upper, lower, left, andright positions of the considered pixel. When a pixel corresponding to agreen component is a considered pixel, color information of the greencomponent of the considered pixel can be used as it is.

Thereafter, the image processing portion 11 detects the position wherethe red component is shifted from the considered pixel in accordancewith the color shift amount of the red component to the green component,and calculates color information of the red component of the consideredpixel by performing linear interpolation with reference to colorinformation of red components arranged in the local region (for example,4×4 pixels) that contains the shifted position. Likewise, the imageprocessing portion 11 calculates color information of a blue componentof the considered pixel.

When a pixel that corresponds to a red component and is placed atcoordinates [i, j] is a considered pixel, and a red component to befocused on the coordinates [i, j] is shifted to a position representedby a black circle shown in FIG. 6 (at an intermediate position ofcoordinates [i+2, j−3] and coordinates [i+3, j−3]), color information ofthe red component of the considered pixel can be calculated with colorinformation (equivalent to R[i+2, j−4], R[i+4, j−4], R[i+2, j−2], R[i+4,j−2]) of red components in a local region 1 shown in FIG. 6. On theother hand, when a blue component to be focused on coordinates [i, j] isshifted to a position represented by a white circle shown in FIG. 6 (atan intermediate position of coordinates [i−3, j+3] and coordinates [i−2,j+3]), color information of a green component of the considered pixelcan be calculated with color information (equivalent to B[i−3, j+3],B[i−1, j+3] of blue components in a local region 2 shown in FIG. 6.

In other words, a local region that contains color informationreferenced when calculating color information of a red component or ablue component is moved from a local region that contains a consideredpixel. The moved amount of a local region is varied in accordance withthe color shift amount corresponding to the position of a consideredpixel.

Thereafter, the image processing portion 11 adds with weight the colorinformation of each color component of the considered pixel with apredetermined weighting coefficient, thereby generating a Y component.

As described above, according to the third embodiment, since colorinformation of a color component that has color shift can be calculatedwith reference to color information placed in a local region moved inaccordance with a color shift amount, a Y component can be accuratelygenerated without performing a filtering process for correcting colorshift.

Alternatively, such a Y component generating process as described abovecan be accomplished by a filtering process using a coefficient filter.

For example, for a considered pixel in which color shift takes place asshown in FIG. 6, a filtering process using a coefficient filter shown inFIG. 7( b) may be performed instead of a conventional coefficient filtershown in FIG. 7( a). The coefficient filter shown in FIG. 7( b) isstructured in such a manner that a center position of a green componentfilter is not moved whereas center positions of a red component filterand a blue component filter are moved in accordance with a color shiftamount.

In addition, color shift amounts of adjacent pixels hardly change. Whenimage data is divided into a plurality of blocks (as shown in FIG. 5),color shift amounts of pixels in the same block are almost the same.

Thus, when the image processing portion 11 pre-records a color shiftamount of a representative pixel in each block and references thepre-recorded color shift amount, it is not necessary for the imageprocessing portion 11 to calculate a color shift amount for eachconsidered pixel. When the calculation for obtaining a color shiftamount for each considered pixel is omitted, a calculation for obtainingthe distance from the center of the image to a considered pixel can beomitted. Thus, a Y component can be quickly and accurately generated.

By pre-recording a value of a coefficient filter for a representativepixel in each block, a Y component can be accurately and quicklygenerated when a Y component generating process is performed by afiltering-process using a coefficient filter.

When image data is divided into 5×7 blocks as shown in FIG. 5, it isrecommended that: a filtering process for block 1 is performed with acoefficient filter shown in FIG. 7( a); a filtering process for block 4is performed with a coefficient filter shown in FIG. 7( b); and afiltering process for block is performed with a coefficient filter shownin FIG. 8.

Alternatively, it is not necessary to pre-record values of coefficientfilters of all blocks. For example, only values of coefficient filtersof blocks at immediately upper, lower, left, and right positions thatare adjacent to a block containing the center of an image may bepre-recorded. Values of coefficient filters of another blocks may begenerated by combining four pre-recorded values of coefficient filtersand correcting the resultant combined value in accordance with thedistance from the center of the image to each block.

According to the third embodiment, the Y component generating processthat does not need to perform the image analyzing process was described.Alternatively, similarity of a considered pixel in a predetermineddirection may be determined. In accordance with the determined result,the color information in each local region as described above may beselectively added with weight so as to calculate color information ofeach color component.

When a pixel corresponding to a red component (or a blue component) is aconsidered pixel and it has been determined that the similarity of theconsidered pixel is strong in the vertical direction, color informationof a green component of the considered pixel can be calculated byaveraging color information of green components placed at theimmediately upper and lower positions of the considered pixel. Such aprocess can be accomplished by performing a filtering process usingcoefficient filters as shown in FIG. 9 instead of the coefficientfilters shown in FIG. 7( b).

According to the third embodiment, color shift is suppressed by changingthe relative locations of a considered pixel and pixels from which colorinformation is referenced when generating a Y component, in accordancewith the position of the considered pixel in the image. Such suppressionof color shift can be performed in a Y component generating process forimage data (in which each pixel has color information of three colors R,G, and B in RGB color system) generated through a multi-CCD type imagesensor.

Likewise, a technology for changing the relative locations of aconsidered pixel and pixels from which color information is referencedwhen performing an image process can be applied to an interpolatingprocess for generating color information of a green component for apixel that lacks color information of a green component. The technologycan also be applied to a similarity factor calculating process. As aresult, color shift can be suppressed.

In particular, like the foregoing Y component generating process, aninterpolating process can be accomplished by moving the center positionof a particular color component filter of conventional coefficientfilters in accordance with a color shift amount of the position of theconsidered pixel in the image, and performing a filtering process withthe obtained coefficient filters.

When it has been determined that similarity in the vertical direction isstrong and color shift of a red component to a green component takesplace in a pixel corresponding to a red component, as shown in FIG. 6, agreen component of the pixel can be generated by a filtering processusing coefficient filters shown in FIG. 10( b) instead of a conventionalcoefficient filter shown in FIG. 10( a).

In the foregoing example, an interpolating process in the verticaldirection was described. According to the present invention, however, aninterpolating process in the horizontal direction can also be performedin the same manner.

In the foregoing first to third embodiments, an image processaccomplished by the image processing portion 11 of the electronic camera1 was described. The image process can be accomplished by an imageprocessing device composed of for example a personal computer. In otherwords, in the electronic camera 1, image data may be recorded in thememory card 16 without being through the image processing portion 11,and the recorded image data may be read from the memory card 16 tosubject to an image process, in the same manner as the image processingportion 11, by the image processing device composed of a personalcomputer and the like.

However, when information that represents a characteristic of aphotographic optical system that generates an image cannot be obtainedin the image processing device, the ratio of k1 to k2 of Formula 8according to the first and second embodiments and the color shift amountaccording to the third embodiment can be designated in accordance with acharacteristic of conventional chromatic aberration of magnification.

In the foregoing embodiments, the present invention is applied to anelectronic camera. However, the present invention is not limited to theelectronic camera. For example, the image processing device according tothe present invention may be disposed in a scanner device such as a filmscanner.

Moreover, when an operation of the foregoing image processing portion iscoded, an image processing program that causes a computer to execute animage process can be obtained.

Although the present invention has been shown and described with respectto a best mode embodiment thereof, it should be understood by thoseskilled in the art that the foregoing and various other changes,omissions, and additions in the form and detail thereof may be madetherein without departing from the spirit and scope of the presentinvention.

Industrial Utilization

As described above, according to the image processing device of thepresent invention, an image process can be performed with suppressingthe influence of color shift due to chromatic aberration ofmagnification of a photographic optical system.

1. An image processing device, comprising: a similarity factor calculating portion locally comparing pixels that compose an image so as to calculate a similarity factor of the pixels in a predetermined direction; an analyzing portion analyzing structure of the image in accordance with the similarity factor calculated by the similarity factor calculating portion, wherein the similarity factor calculating portion calculates the similarity factor of the pixels in accordance with positions of the pixels in the image; the similarity factor includes a value determined by performing a weighted addition of “a similarity factor element in same colors obtained by comparing pixels of same colors” and “a similarity factor element in different colors obtained by comparing pixels of different colors”; and the similarity factor calculating portion changes a weighting coefficient for the weighted addition in accordance with the positions of the pixels whose similarity factor is to be calculated, and calculates the similarity factor thereof.
 2. The image processing device as set forth in claim 1, wherein the similarity factor calculating portion changes the weighting coefficient in accordance with distance from “the center of the image” to “the pixels whose similarity factor is to be calculated”.
 3. The image processing device as set forth in claim 2, wherein the similarity factor calculating portion changes the weighting coefficient so that weight of the similarity factor element in different colors is smaller at a peripheral part of the image than at the center part of the image.
 4. The image processing device as set forth in claim 2, wherein the similarity factor calculating portion divides the image into a plurality of regions and changes the weighting coefficient for each of the divided regions.
 5. The image processing device as set forth in claim 1, wherein the similarity factor calculating portion changes the weighting coefficient in accordance with a characteristic of color shift of an optical system for generating the image.
 6. An image processing device, comprising: a similarity factor calculating portion locally comparing pixels that compose an image so as to calculate a similarity factor of the pixels in a predetermined direction; an analyzing portion analyzing structure of the image in accordance with the similarity factor calculated by the similarity factor calculating portion, wherein the similarity factor calculating portion calculates the similarity factor of the pixels in accordance with positions of the pixels in the image; the similarity factor calculating portion calculates a similarity factor including a weighted addition value of “a similarity factor element in same colors obtained by comparing pixels of same colors” and “a similarity factor element in different colors obtained by comparing pixels of different colors” in a region containing the center of the image; and the similarity factor calculating portion calculates a similarity factor that does not contain the similarity factor element in different colors, but contains the similarity factor element in same colors in a region other than that containing the center of the image.
 7. An image Processing device, comprising: a similarity factor calculating portion locally comparing pixels that compose an image so as to calculate a similarity factor of the pixels in a predetermined direction; an analyzing portion analyzing structure of the image in accordance with the similarity factor calculated by the similarity factor calculating portion, wherein the similarity factor calculating portion calculates the similarity factor of the pixels in accordance with positions of the pixels in the image; and the similarity factor calculating portion changes relative location of “a considered pixel whose similarity factor is to be calculated” and “a reference pixel that is referenced when calculating the similarity factor” in accordance with position of the considered pixel.
 8. The image processing device as set forth in claim 7, wherein the similarity factor calculating portion changes the relative location in accordance with distance from the center of the image to the considered pixel.
 9. The image processing device as set forth in claim 8, wherein the similarity factor calculating portion divides the image into a plurality of regions and changes the relative location for each of the regions.
 10. The image processing device as set forth in claim 7, wherein the similarity factor calculating portion changes the relative location in accordance with a characteristic of color shift of an optical system for generating the image.
 11. An image processing device, comprising: a similarity factor calculating portion locally comparing pixels that compose an image so as to calculate a similarity factor of the pixels in a predetermined direction; and a pixel value calculating portion referencing the pixels that compose the image so as to obtain a value of a new pixel, wherein the similarity factor calculating portion calculates the similarity factor of the pixels in accordance with positions of the pixels in the image, and the pixel value calculating portion changes relative location of “a considered pixel from which the new value is to be calculated” and “a reference pixel that is referenced when calculating the new value” in accordance with position of the considered pixel and the similarity factor, and calculates the new value of the considered pixel.
 12. The image processing device as set forth in claim 11, wherein the pixel value calculating portion changes the relative location predetermined in accordance with “distance from the center of the image to the considered pixel”, according to the similarity factor.
 13. The image processing device as set forth in claim 12, wherein the pixel value calculating portion divides the image into a plurality of regions and changes the relative location predetermined for each of the regions in accordance with the similarity factor.
 14. The image processing device as set forth in claim 11, wherein the pixel value calculating portion changes the relative location predetermined in accordance with “a characteristic of color shift of an optical system for generating the image”, according to the similarity factor.
 15. The image processing device as set forth in claim 14, wherein: when the new value contains an element composed of a value of a color that has color shift and the element is calculated, the pixel value calculating portion selects, as the reference pixel, a pixel in a local region selected in accordance with a direction of the color shift and an amount of the color shift; and the pixel value calculating portion adds with weight the selected reference pixel in accordance with the similarity factor so as to calculate the new value.
 16. A computer readable medium storing an image processing program, causing a computer to perform the functions of: locally comparing pixels that compose an image with a similarity factor calculating portion so as to calculate a similarity factor of the pixels in a predetermined direction; and referencing the pixels that compose the image with a pixel value calculating portion so as to obtain a value of a new pixel, wherein the similarity factor calculating portion calculates the similarity factor of the pixels in accordance with positions of the pixels in the image, and the pixel value calculating portion changes relative location of “a considered pixel from which the new value is to be calculated” and “a reference pixel that is referenced when calculating the new value” in accordance with position of the considered pixel and the similarity factor, and calculates the new value of the considered pixel. 